Dual Attention Networks for Visual Reference Resolution in Visual Dialog

Visual dialog (VisDial) is a task which requires an AI agent to answer a series of questions grounded in an image. Unlike in visual question answering (VQA), the series of questions should be able to capture a temporal context from a dialog history and exploit visually-grounded information. A problem called visual reference resolution involves these challenges, requiring the agent to resolve ambiguous references in a given question and find the references in a given image. In this paper, we propose Dual Attention Networks (DAN) for visual reference resolution. DAN consists of two kinds of attention networks, REFER and FIND. Specifically, REFER module learns latent relationships between a given question and a dialog history by employing a self-attention mechanism. FIND module takes image features and reference-aware representations (i.e., the output of REFER module) as input, and performs visual grounding via bottom-up attention mechanism. We qualitatively and quantitatively evaluate our model on VisDial v1.0 and v0.9 datasets, showing that DAN outperforms the previous state-of-the-art model by a significant margin.Comment: EMNLP 201

PROGrasp: Pragmatic Human-Robot Communication for Object Grasping

Interactive Object Grasping (IOG) is the task of identifying and grasping the desired object via human-robot natural language interaction. Current IOG systems assume that a human user initially specifies the target object's category (e.g., bottle). Inspired by pragmatics, where humans often convey their intentions by relying on context to achieve goals, we introduce a new IOG task, Pragmatic-IOG, and the corresponding dataset, Intention-oriented Multi-modal Dialogue (IM-Dial). In our proposed task scenario, an intention-oriented utterance (e.g., "I am thirsty") is initially given to the robot. The robot should then identify the target object by interacting with a human user. Based on the task setup, we propose a new robotic system that can interpret the user's intention and pick up the target object, Pragmatic Object Grasping (PROGrasp). PROGrasp performs Pragmatic-IOG by incorporating modules for visual grounding, question asking, object grasping, and most importantly, answer interpretation for pragmatic inference. Experimental results show that PROGrasp is effective in offline (i.e., target object discovery) and online (i.e., IOG with a physical robot arm) settings.Comment: 7 pages, 6 figure

PGA: Personalizing Grasping Agents with Single Human-Robot Interaction

Language-Conditioned Robotic Grasping (LCRG) aims to develop robots that ground and grasp objects based on natural language instructions. While robots capable of recognizing personal objects like "my wallet" can interact more naturally with non-expert users, current LCRG systems primarily limit robots to understanding only generic expressions. To this end, we introduce a task scenario GraspMine with a novel dataset that aims to locate and grasp personal objects given personal indicators via learning from a single human-robot interaction. To address GraspMine, we propose Personalized Grasping Agent (PGA), that learns personal objects by propagating user-given information through a Reminiscence-a collection of raw images from the user's environment. Specifically, PGA acquires personal object information by a user presenting a personal object with its associated indicator, followed by PGA inspecting the object by rotating it. Based on the acquired information, PGA pseudo-labels objects in the Reminiscence by our proposed label propagation algorithm. Harnessing the information acquired from the interactions and the pseudo-labeled objects in the Reminiscence, PGA adapts the object grounding model to grasp personal objects. Experiments on GraspMine show that PGA significantly outperforms baseline methods both in offline and online settings, signifying its effectiveness and personalization applicability on real-world scenarios. Finally, qualitative analysis shows the effectiveness of PGA through a detailed investigation of results in each phase.Comment: 7 pages, under revie

Risk stratification of symptomatic brain metastases by clinical and FDG PET parameters for selective use of prophylactic cranial irradiation in patients with extensive disease of small cell lung cancer

Purpose: To identify risk factors for developing symptomatic brain metastases and evaluate the impact of prophylactic cranial irradiation (PCI) on brain metastasis-free survival (BMFS) and overall survival (OS) in extensive disease small cell lung cancer (ED-SCLC). Materials and methods: Among 190 patients diagnosed with ED-SCLC who underwent FDG PET/CT and brain Magnetic Resonance Imaging (MRI) prior to treatment, 53 (27.9%) received PCI while 137 (72.1%) did not. Prognostic index predicting a high risk of symptomatic brain metastases was calculated for the group without receiving PCI (observation group, n = 137) with Cox regression model. Results: Median follow-up time was 10.6 months. Multivariate Cox regression showed that the following three factors were associated with a high risk of symptomatic brain metastases: the presence of extrathoracic metastases (p = 0.004), hypermetabolism of bone marrow or spleen on FDG PET (p < 0.001), and high neutrophil-to-lymphocyte ratio (p = 0.018). PCI significantly improved BMFS in high-risk patients (1-year rate: 94.7% vs. 62.1%, p = 0.001), but not in low-risk patients (1-year rate: 100.0% vs. 87.7%, p = 0.943). However, PCI did not improve OS in patients at high risk for symptomatic brain metastases (1-year rate: 65.2% vs. 50.0%, p = 0.123). Conclusion: Three prognostic factors (the presence of extrathoracic metastases, hypermetabolism of bone marrow or spleen on FDG PET, and high neutrophil-to-lymphocyte ratio) were associated with a high risk of symptomatic brain metastases in ED-SCLC. PCI was beneficial for patients at a high risk of symptomatic brain metastases in terms of BMFS, but not OS. Thus, selective use of PCI in ED-SCLC according to the risk stratification is recommended. (C) 2020 Elsevier B.V. All rights reserved.

Fully automated identification of brain abnormality from whole-body FDG-PET imaging using deep learning-based brain extraction and statistical parametric mapping

Background The whole brain is often covered in [18F]Fluorodeoxyglucose positron emission tomography ([18F]FDG-PET) in oncology patients, but the covered brain abnormality is typically screened by visual interpretation without quantitative analysis in clinical practice. In this study, we aimed to develop a fully automated quantitative interpretation pipeline of brain volume from an oncology PET image. Method We retrospectively collected 500 oncologic [18F]FDG-PET scans for training and validation of the automated brain extractor. We trained the model for extracting brain volume with two manually drawn bounding boxes on maximal intensity projection images. ResNet-50, a 2-D convolutional neural network (CNN), was used for the model training. The brain volume was automatically extracted using the CNN model and spatially normalized. For validation of the trained model and an application of this automated analytic method, we enrolled 24 subjects with small cell lung cancer (SCLC) and performed voxel-wise two-sample T test for automatic detection of metastatic lesions. Result The deep learning-based brain extractor successfully identified the existence of whole-brain volume, with an accuracy of 98% for the validation set. The performance of extracting the brain measured by the intersection-over-union of 3-D bounding boxes was 72.9 ± 12.5% for the validation set. As an example of the application to automatically identify brain abnormality, this approach successfully identified the metastatic lesions in three of the four cases of SCLC patients with brain metastasis. Conclusion Based on the deep learning-based model, extraction of the brain volume from whole-body PET was successfully performed. We suggest this fully automated approach could be used for the quantitative analysis of brain metabolic patterns to identify abnormalities during clinical interpretation of oncologic PET studies.This research was supported by the National Research Foundation of Korea (NRF-2019R1F1A1061412 and NRF2019K1A3A1A14065446). This work was supported by the Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (Project Number: 202011A06) and Seoul R&BD Program (BT200151)

Counting independent sets in Riordan graphs

The notion of a Riordan graph was introduced recently, and it is a far-reaching generalization of the well-known Pascal graphs and Toeplitz graphs. However, apart from a certain subclass of Toeplitz graphs, nothing was known on independent sets in Riordan graphs. In this paper, we give exact enumeration and lower and upper bounds for the number of independent sets for various classes of Riordan graphs. Remarkably, we offer a variety of methods to solve the problems that range from the structural decomposition theorem to methods in combinatorics on words. Some of our results are valid for any graph

A pan-cancer analysis of the clinical and genetic portraits of somatostatin receptor expressing tumor as a potential target of peptide receptor imaging and therapy

Abstract Purpose Although somatostatin receptor (SST) is a promising theranostic target and is widely expressed in tumors of various organs, the indication for therapies targeting SST is limited to typical gastroenteropancreatic neuroendocrine tumors (NETs). Thus, broadening the scope of the current clinical application of peptide receptor radiotherapy (PRRT) can be supported by a better understanding of the landscape of SST-expressing tumors. Methods SST expression levels were assessed in data from The Cancer Genome Atlas across 10,701 subjects representing 32 cancer types. As the major target of PRRT is SST subtype 2 (SST2), correlation analyses between the pan-cancer profiles, including clinical and genetic features, and SST2 level were conducted. The median SST2 expression level of pheochromocytoma and paraganglioma (PCPG) samples was used as the threshold to define high-SST2 tumors. The prognostic value of SST2 in each cancer subtype was evaluated by using Cox proportional regression analysis. Results We constructed a resource of SST expression patterns associated with clinicopathologic features and genomic alterations. It provides an interactive tool to analyze SST expression patterns in various cancer types. As a result, eight of the 31 cancer subtypes other than PCPG had more than 5% of tumors with high-SST2 expression. Low-grade glioma (LGG) showed the highest proportion of high-SST2 tumors, followed by breast invasive carcinoma (BRCA). LGG showed different SST2 levels according to tumor grade and histology. IDH1 mutation was significantly associated with high-SST2 status. In BRCA, the SST2 level was different according to the hormone receptor status. High-SST2 status was significantly associated with good prognosis in LGG patients. High-SST2 status showed a trend for association with poor prognosis in triple-negative breast cancer subjects. Conclusion A broad range of SST2 expression was observed across diverse cancer subtypes. The SST2 expression level showed a significant association with genomic and clinical aspects across cancers, especially in LGG and BRCA. These findings extend our knowledge base to diversify the indications for PRRT as well as SST imaging